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Label zones in accordance with a schema. 



Associate mark-up language tags with the labeled zones 
responsive to the identified layout information and a model 
file to generate structured mark-up language documents. 
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(57) Abstract: A method, system, apparatus, 
and graphical user interface (GUI) for generating 
structured document files &om a document image is 
disclosed. Structured document files are generated 
by segmenting the document image into one or 
more zones containing respective text images, 
converting the respective text images to digital text, 
automatically identifying layout information for each 
of the one or more zones, labeling each of the one 
or more zones in accordance with a schema, and 
automatically associating mark-up language tags with 
the labeled zones to generate the stnictured document 
files responsive to the identified layout information 
and a model file. 
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METHOD, SYSTEM, AND APPARATUS FOR GENERATING STRUCTURED DOCUMENT FILES 

CROSS REFERENCE TO RELATED APPLICATIONS 

[0001] This application is reiated to and claims tlie benefit of U.S. Provisional 
Application No. 60/404,581, filed August 20, 2002, for ''A System for Generating 
Structured Document." In addition, this application is related to U.S. Patent Application 
No. 10/293,859 , filed November 13, 2002, for "'Document Classification and Labeling 
Using 1-ayout Graph Matching." 

HELD OF THE INVENTION 

[0002] The present Invention relates to the field of structured, languages and, more 
particularly, to the generation of structured language document files from document 
images. 

BACKGROUND OF THE INVENTION 

[0003] Structured languages such as extensible mark-up language (XML) enable 
the creation of structured document files that are easily searchable and are viewable 
across multiple platforms, e.g., on a desktop computer and on a cellular telephone. For 
example, a structured document file retrieved via a global information network (e.g., the 
Internet) can be viewed in full on a desktop computer and can be viewed as text only on a 
cellular telephone. It is often desirable to convert existing hard copy documents or images 
of documents to structured document files to facilitate searching and displaying these 
documents. Accordingly, methods, systems, and apparatus for converting documents to 
structured document files are useful. 

[0004] Existing documents are typically converted to structured document files by 
scanning the documents and automatically converting the text within the scanned 
documents to digital text using optical character recognition (OCR) software. The scanned 
and converted documents are then formatted, either manually or using proprietary data 
structures, to add mark-up language tags. Often, several different software packages are 
employed to perform each of these steps. These methods for generating structured 
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document files tend to be inflexible, time consuming, and/or difficult to use. In addition, 
tiie original fonnattlng of tfie document Is often lost, e.g., font sizes, emphasis, etc., 
maldng them more difficult to read when they are displayed. 

[0005] Accordingly, methods, systems, and apparatus for converting existing 
documents to structured document files are needed that are not subject to the above 
limitations. The present Invention fulfills this need among others. 

SUMMARY OF THE INVENTION 

[0006] The present invention is a method, system, and apparatus for generating 
structured document files from document images. Structured document flies are 
generated by segmenting the document image into one or more zones containing 
respective text images, converting the respective text images to digital text, automatically 
identifying layout information for each of the one or more zones, labeling each of the one 
or more zones in accordance with a schema, and automatically associating mark-up 
language tags with the labeled zones to generate the structured document files responsive 
to the identified layout information and a model file. 

BFQEF DESCRIPTION OF THE DRAWINGS 

[0007] Figure 1 is a blocl< diagram that conceptually represents an exemplary 
system architecture for generating structured document files from document Images in 
accordance with the present invention; 

[0008] Rgure 2 Is a flow chart of exemplary steps for generating structured 
document files from document images in accordance with the present invention; 

[0009] Figure 3 is an exemplary graphical user interface (GUI) for assisting a user 
in generating structured document files in accordance with the present invention; and 

[0010] Figure 4 is an exemplary document from which structured document files are 
generated in accordance with the present invention. 
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DETAILED DESCRIPTEON OF THE INVENTION 

[0011] FIG. 1 is a conceptual representation of an exempiar/ system architecture 
100 for generating structured document flies from document images in accordance with 
the present invention. One or more bloclcs within the illustrated system architecture 100 
can be performed by the same piece of hardware or module of software. It should be 
understood that embodiments of the present Invention may be implemented in hardware, 
software, or a combination thereof. In such embodiments, the various component and 
steps described below would be implemented in hardware and/or software. 

[0012] In the illustrated system architecture 100, an electronic image of a 
document (the '^document Image") Is applied to a document processor 102. In certain 
exemplary embodiments, the document Image is generated by scanning a physical 
document using conventional scanning techniques. In certain other exemplary 
embodiment, the document Image is supplied in an electronic format such as a Tagged 
Image File Format (tiff) file. Joint Photographic Experts Group (jpeg) file, or other such file. 
In these embodiments, a format converter (not shown) may be used to convert the 
document Image into a format compatible with the present invention. Suitable document 
images and format converters for use with the present Invention will be readily apparent to 
those of skill in the related arts. 

[0013] The document processor 102 processes the document image in preparation 
for labeling and generating the structured document file(s), which actions are described In 
greater detail below. The illustrated document processor 102 Includes a segmenter 104, a 
text converter 106, and a zone and text editor 108. The segmenter 104 segments the 
document Image Into zones containing text or images. For example, the segmenter 104 
may create a zone containing the title of a document, a zone containing a paragraph 
within the document, and a zone containing a figure. In addition, the segmenter 104 
determines layout information for the zones. For example, the font size and the position of 
the zone on the document. A suitable segmenter for use with the present invention will be 
readily apparent to those of skill in the art of Image processing. Additional information 
regarding segmenters can be found in commonly assigned U.S. Patent Nos. 5,892,843 and 
6, 327,388 to Zhou et al. entitled '"Title, Caption and Photo Extraction from Scanned 
Document Images'' and '"Identification of Logos from Document Images," respectively. 
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[0014] In an exemplary embodiment, the segmenter 104 identifies wliich zones 
contain text images and whicli zones contain figures. In certain exemplary embodiments, 
eacli zone Is displayed in a color that represents the type of information within that zone. 
For example, text image may be displayed In one color, e.g., red, and non-text images 
such as tables and figures may be displayed In another color, e.g., green. In certain other 
exemplary embodiments, the zones may be distinguished in other ways such as with a 
border having a different color or pattern. 

[0015] The text converter 106 converts the text images of the zones to digital text, 
i.e., text which Is searchable and editable. For example, the text converter may convert 
the letters with the text images to their ASCII equivalent. In a exemplary embodiment, 
the text converter is a conventional optical character recognition (OCR) software tool. 
Suitable text converters for use with the present invention will be readily apparent to those 
of skill in the art of image processing. 

[0016] The zone and text editor 108 edits the zones and the digital text. In an 
exemplary embodiment, the zone and text editor 108 may add zones, delete zones, or 
change the size of individual zones responsive to user inputs. For example, a user may 
enlarge a zone containing a portion of a document title to Include the entire title. In an 
exemplary embodiment, layout Information associated with a zone is updated in 
accordance with the changes to the zones. The zone and text editor 108 also may change 
the digital text responsive to user inputs. For example, misspelled words may be corrected 
by a user. In an exemplary embodiment, the zone and text editor 108 receives user Inputs 
via a graphical user interface, which is described in detail below. Suitable zone and text 
editors for use with the present invention will be readily apparent to those of sl<ill In the art 
of Image processing. 

[0017] The document, as processed by the document processor 102, is applied to a 
model selector 110. The model selector 110 selects a previously developed model file, 
described below, having features that resemble features of the document. In an 
exemplary embodiment, the model selector selects the model file from a plurality of 
previously developed model files. Each of the model files references a schema, which 
describes the structure of a document that contains valid semantics (e.g. title, author, 
abstract etc. for a document such as a technical paper) and includes physical 
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charac±eristics for the elements of the schema and their spatial relationships relative to 
one another. 

[0018] In an exemplar/ embodiment, the model file is selected by a user, e.g., via 
the graphical user Interface (GUI) described below. In an altemative exemplary 
embodiment, the model selector 110 compares features of the processed document image 
to stored features of previously developed model files to automatically select a model file. 
In certain exemplary embodiments, a costing technique is employed with a cost assigned 
to each feature and lower costs representing a higher level of resemblance. In accordance 
with this embodiment, a comparison cost is determined for each available model file 
compared to the document image and the model file with the lowest cost is selected. A 
method for automatically selecting a model file by matching features is described In 
commonly assigned U.S. Patent Application No. 10/293,859 , filed November 13, 2002, for 
""Document Classification and Labeling Using layout Graph Matching." having at least one 
common inventor (referred to herein as the "^Document Classification and Labeling 
Application"). 

[0019] A schema editor 112 edits the schema. In an exemplary embodiment, the 
schema is retrieved based on a reference to the schema in the model file. In an 
altemative exemplary embodiment, the schema may be referenced by a user, e.g., via the 
GUI described below. The schema editor 112 may be used to add or remove elements 
from the schema responsive to user inputs. In an exemplary embodiment, the schema 
editor 112 Is displayed In a tree-view and the user inputs are received via the GUI 
described below. A suitable schema editor will be readily apparent to those of sl<lll in the 
related arts. 

[0020] A model developer 114 develops the models for use by the model selector 
110. In an exemplary embodiment, the model developer 114 develops the model by 
processing document samples. In certain exemplary embodiment, the model developer 
114 develops the model responsive to user inputs. If the schema Is changed by the 
schema editor 112, the model developer 114 needs to develop a new model in accordance 
with the new schema that accommodates the new relations. A suitable model developer 
for use with the present invention is described in the Document Classification and Labeling 
Application. 
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[0021] In an exemplary embodiment, models are developed at a system level. 
When developed at the system level, a user's edit and correction activities of logical 
labeling results are monitored. An automatic model learning process updates the 
document model through a feedback loop based on user modified results. In an 
alternative exemplary embodiment, models are developed at the user level. When 
developed at the user level, a GUI tool is provided to allow a more knowledgeable user to 
manually create a new model from a set of known samples. 

[0022] The document, as processed by the document processor 102, is also applied 
to a labeler 116. The labeler 116 applies labels to the zones defined by the document 
processor 102 in accordance with the schema. For example, the labeler may label a zone 
containing the title of the document with the element "^tltle." In an exemplary 
embodiment, the labeler applies labels to the zones responsive to a document model 
selected by the model selector 110. 

[0023] In an exemplary embodiment, the labeler 116 automatically labels the zones 
using a layout graph technique. An exemplary layout graph represents each schema 
element associated with a selected model file and its spatial relationships to one or more of 
the other schema elements and another exemplar/ layout graph represents each zone in a 
document image and its spatial relationship to one or more of the other zones. In the 
exemplary embodiment, a document image is compared to a selected model by the layout 
graphs using a known global scale over total cost matching technique. Because some 
elements in a document may correspond to multiple zones, multiple zones may match the 
same element. A suitable layout graph technique for use with the present invention, from 
which one skilled In the art can develop a suitable labeler 116, Is described in the 
Document Classification and Labeling Application. 

[0024] A label editor 118 enables manual editing of the labeled zones. In an 
exemplar/ embodiment, the label editor 118 updates the labels on zones applied 
automatically by the labeler 116 responsive to user inputs. For example, if the labeler 116 
labeled a zone containing the title of the document with the element ^^author," the label 
editor can be used to change the label of that zone to the correct element, i.e., "title." In 
an alternative exemplar/ embodiment, the label editor 118 labels each of the zones 
manually responsive to user inputs. In an exemplary embodiment, the label editor 118 
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receives user inputs via the GUI described below. A suitable label editor 118 for use witii 
the present Invention will be readily apparent to those of skill In the art of image 
processing. 

[0025] A structured document generator 120 generates structured document files 
responsive to layout information associated with the zones, labeling results, and the 
selected model file. In an exemplary embodiment, the structured document generator 120 
generates an extensible mark-up language (XML) file and a extensible style-sheet 
language (XSL) file for each document image that It processes. The XML file represents 
the document structure and the XSL file represents the document layout. In an exemplar/ 
embodiment, the XSL file may represent layout information such as font type and size, 
font color, and zone coordinates. 

[0026] To develop the XML file, the exemplary structured document generator 120 
receives layout information from the document processor 102 and labeling results from the 
iabeler 116. In an exemplary embodiment, the layout information contains the number of 
zones within the document, identification numbers for each zone, and the location of each 
zone. In addition, the structured document generator 120 receives digital text for each 
zone containing a text image from the document processor 102. In certain exemplary 
embodiments, the document processor 102 develops a layout file that includes the layout 
information and the digital text. In these embodiments, the document processor 102 
passes the layout file to the structured document generator 120 for processing. In certain 
other exemplary embodiments, the digital text is included within the labeling results. 

[0027] The exemplary structured document generator 120 uses the labeling results 
to match each zone to the appropriate schema elements. The structured document 
generator 120 then combines the layout file and the labeling results In a manner that will 
be readily apparent to those skilled In the art of computer programming to generate the 
XML file. A portion of an exemplary XML file Is depicted In Table 6 below. 

[0028] In certain exemplary embodiments for generating the XML file, the 
structured document generator 120 also receives the model file, which contains the 
schema, from the model selector 110. The document generator 120 may then validate the 
labeling results by comparing the labeling results to the schema to verify that each label of 
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the labeling results corresponds to a schema elennent. In addition, the structured 
document generator 120 may use the model file to Incorporate a complete document tree 
structure Into the XML file. For example, the element ^'name" may contain two sub- 
elements, e.g., first name and last name. In this embodiment, the structure for the sub- 
elements may be included in the XML file. The Incorporation of the document tree 
structure into the XML file will be readily apparent to those of skill In the art of computer 
programming. Also, the structured document generator 120 may use the model file to 
match individual elements to corresponding layout information in the layout file, e.g., 
using zone coordinates contained in the layout file and in the model file. 

[0029] To develop the XSL file, the exemplary structured document generator 120 
receives the layout Information from the document processor 102, the labeling results from 
the labeler 116, and the model file from the model selector 110. Pseudo code to direct 
element processing to generate the XSL file is depicted in Table 1. 

TABLE 1. 

Start root of tree 
Repeat nodes 
If leaf node; no child node 
If this node matches multiple zones 

output xsl template using <xsl:for-each> 
else 

output xsl template using <xsl: template match > 

get next node 
Else; has child node 

get child node 
Endif 

The pseudo code depicted in Table 1 illustrates the processing of the elements by the 
structured document generator 120. In a tree view representation of the schema, each 
element of the schema is represented as a node. Each node can have one or more child 
nodes. For example, a logical element ''author" can have two child nodes, e.g., 'Mast 
name" and "first name", and it can have multiple instances to refiect multiple authors. A 
node can also be a leaf node, which indicates there is no branches from this node, such as 
^'first name" or ''last name." Processing continues until all elements/nodes are processed. 
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[O03O] For each element processed by the structured document generator 120, the 
structured document generator 120 matches the element to corresponding layout 
Information In the layout file, e.g., using zone coordinates contained in the layout file and 
In the model file. The structured document generator 120 then combines the element with 
the corresponding layout information to generate the XSL file in a manner that will readily 
apparent to those of sldll In the art of computer programming. 

[0031] In certain exemplar/ embodiments, a layer concept associated with the 

hypertext mark-up language (HTML) preserves the original layout, e.g., using 
<DIV></DIV> tags in the XSL file. Each layer enclosed within the <DIV></DIV> tags is 
independent of every other layer. Thus, a zone in one layer has no effect on the position 
of a zone In another layer when the zones are displayed on a Icnown web browser (not 
shown). Accordingly, a zone may be assigned coordinates with respect to a common origin 
for display on a web browser without affecting the positioning of any other zone. In 
addition, each zone can have its own style, e.g., font size, type, and color. In an 
exemplary embodiment, each zone is assigned to a different layer. The original 
coordinates for each zone are then used to develop display coordinates in a known manner 
to display the zone on a web browser. Since the original coordinates for the zones are 
used to position the zones, the zones are referenced to a common origin, and the zones do 
not affect the position of zones in other layers, the position of the zones when displayed on 
a web browser will at least partially match the original layout of the original document 
image when all layers are displayed. Style information such as font size may also be 
included to increase the resemblance between the displayed document and the original 
document image. A portion of an exemplary XSL file is depicted in Table 7 below. 

[0032] In certain exemplary embodiments, one or more of the zones may contain 
non-text images (not shown) that are not converted to digital text such as graphs, 
pictures, etc. In an exemplary embodiment, for each zone containing a non-text image 
the structured document generator 120 generates an image file from the portion of the 
original image within a zone. The structured document generator 120 then inserts a link 
to the Image file in the XML file In a manner similar to the insertion of digital text 
described above to generate the XML file. In addition, the structured document generator 
120 generates the XSL file in a similar manner as described above for text images with the 
exception that style information such as font size is not included. 
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[0033] RG. 2 depicts a flow chart 200 of exemplary steps for generating structured 
document files in accordance with the present invention. Processing begins at block 202 
with the segmentation of the document image into zones at block 204. At block 206, text 
Images within the zones are converted to digital text. At block 208, the zones and digital 
text are edited. In an exemplary embodiment, the zones are segmented, digital text Is 
converted, and zones and digital text are edited as described above with reference to the 
segmenter 104, text converter 106, and editor 108, respectively, of FIG. 1. 

[0034] At block 210, layout information for the document image is identified. The 
layout Information includes non-content related features that define the look of the 
document: These features may include, by way of non-limiting example, font size, 
emphasis formatting, positional Information, etc. In an exemplary embodiment the layout 
information is used in the generation of the structured document files such that a 
displayed image of the structured document files retains at least a portion of the original 
layout information associated with the document image. Because the original layout 
Information is maintained, the displayed Images reflect the formatting of the original 
documents, thus making them more easy to read. In an exemplary embodiment, the 
layout information Is identified by the above-described segmenter 104 (FIG. 1). 

[0035] At block 212, the zones are labeled in accordance with a schema and, at 
block 212, mark-up language tags are associated with to the labeled zones to create the 
structured document flies. In an exemplary embodiment, the zones are labeled and the 
tags are associated as described above with reference to the labeler 116 and the 
structured document generator 120, respectively, of RG. 1. 

[0036] FIG. 3 depicts an exemplary graphical user interface (GUI) 300 for use in the 
present Invention. The illustrated GUI 300 includes a tool bar 302, a schema panel 304, 
and a viewing panel 306. The GUI 300 provides an easy to user interface that allows a 
user to generate structured document files from document images. 

[0037] In an exemplary embodiment, a user accesses a workflow menu (not shown) 
by selecting a '"workflow" indicator 308 from the tool bar 302. In certain exemplary 
embodiments, the workflow menu guides the user sequentially through the structured 
document file generation process described above, e.g., segmenting the document image 
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into zones, converting text to digital text, labeling the zones, and generating the 
structured document files. In certain other exemplary embodiments, the user is guided 
through the worlcflow process by a ""workflow" Icon 310, which is described in detail below. 
In certain exemplary embodiments, arrow indicators 311 are available to move bacic and 
forth sequentially through the workflow process. Alternatively, the entire workflow process 
of generating a structured document from a document image Is performed automatically 
by selecting an '^auto execute'' icon 312 in the toolbar 302. 

[0038] The ''workflow'' Icon 310 displays unique Images that correspond to different 
steps of the workflow process. In an exemplary embodiment, the ''workflow" Icon 310 
reflects a next step in the workflow process to guide a user sequentially through the 
process of generating structured document flies from document Images. For example, 
prior to loading a document image, the ""workflow" icon 310 may display the text "'Load 
Image," and after the document image is loaded, but before the document image is 
segmented, the ^^workflow" Icon 310 may display the text ""Segment." Selecting the 
workflow Icon 310 when the text "Load Image" is displayed results In the loading of an 
Image and selecting the "'workflow" icon 310 when the text ""Segment" is displayed results 
in the segmentation of the document image. 

[0039] In an exemplary embodiment, selecting an "Images" icon 314 on the toolbar 
302 initiates a document Image source selection, e.g., via a conventional file open window 
(not shown). A selected document image is then displayed in the viewing panel 306. 

[0040] In an exemplary embodiment, the selection of a document image initiates a 
model file matching routine that identifies a model file for the document Image. From the 
identified model file, a schema is identified for display in the schema panel 304, e.g.. In a 
tree view. Altematively, a user selects the schema manually by selecting a ""schema" Icon 
316 on the toolbar 302. In certain exemplary embodiments, the user changes the 
automatically or manually selected schema by selecting the "schema" icon 316. In certain 
exemplary embodiment, the schema may be updated, e.g., elements may be added or 
removed from the schema, or a new schema may be created using conventional editing 
techniques. Once editing Is complete, the user saves the newly edited (or created) schema 
file. The model matching process, described above, is performed after a new schema is 
saved to select a model corresponding to the new schema. 
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[0041] Document segmentation, text conversion, and labeiing are performed in the 
viewing panel 306. In an exemplary embodiment, tlie document is segmented and text is 
converted responsive to tlie loading of a document Image. In alternative exemplary 
embodiments, the document Is segmented and the text is converted by selecting the 
"^workflow" Icon 310 on the toolbar 302 twice (once to initiate segmentation and once to 
initiate text conversion) or through the workflow menu (not shown) that appears when the 
workflow Indicator 308 Is selected. In accordance with these embodiments, the document 
is segmented into "^meaningful" zones according to physical attributes such as font size, 
spacing, etc. In the Illustrated embodiment, segmented zones are displayed with 
bounding boxes overlaid on the original Image, which can be corrected by the user using 
conventional techniques. There are several features that are available within the viewing 
panel 306. These features Include zoom in/out, zone selection/editing, and zone change 
features. In the illustrated embodiment, text conversion results for Identified text regions 
are also overlaid directly in each zone for easy review and editing using conventional 
techniques. It will be readily apparent to those of skill in the art that segmentation and 
text conversion may be performed concurrently or in two distinct steps. 

[0042] After segmentation and text conversion, labels are added to the segments. 
In an exemplary embodiment, labeling Is initiated through its selection from the workflow 
menu or by selecting the "^"workflow" icon 310. In the illustrated embodiment, the labeling 
results in the display of logical labels on the top left corner of each zone as shown in FIG. 
3. In an exemplary embodiment, the logical labels can be edited In a conventional 
manner, e.g., by 'Yight-cllcking" to display a pull-down menu (not shown) to link and 
unlink the zone to a schema element or by dragging the schema elements from the 
scheme tree to a zone. In certain exemplary embodiments, the labels associated with the 
zones may be saved by selecting a "^SaveUnk" icon 318 on the tool bar 302. 

[0043] After labeling, the structured documents are generated by selecting a 
structured document generation Indicator in the workflow menu, selecting the ^Vorkflow" 
icon 310, or selecting a '^Save XML" Icon 320 on the toolbar 302. In an exemplary 
embodiment, this prompts the creation of two structured document files: an XML file and a 
corresponding XSL file. 
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[0044] The GUI 300 additionally provides an easy to use interface that allows a user 
to train model files. In an exemplary embodiment, a training mode Is entered by selecting 
this mode from the ^^workflow" menu or by selecting a "^Leaml^odel" Icon 322 on the 
toolbar 302. In the training mode, a user edits one or more similar sample documents. 
During editing, the user's edits are monitored and analyzed to develop a model file from 
the sample documents. The new model file can then be used to segment and label 
subsequent documents. 



[0045] HG. 4 depicts a document image 400 to be processed in accordance with the 
present invention. Initially, the document image 400 is scanned using conventional 
scanning software. The illustrated document Image 400 includes several blocks of text 
including a title 402 and author information 404, e.g., name, telephone number, etc. A 
schema for a two-column text document similar In style to the document Image 400 is 
included in Table 2. 



Table 2 

<xsd:schema xmlns:xsd="http://www.w3.org/1999/XMLSchema"> 

<xsd:eiement name="document"> 

<xsd:complexType> 

<xsd:element name="title" type="xsd:strlngV> 
<xsd:element name="leftColumnText" type="xsd:string"/> 
<xsd:element name= "abstract" type="xsd:string7> 
<xsd:element name="author" type="xsd:string"/> 
<xsd:element name="leftHeader" type="xsd:string"/> 
<xsd: element name="page" type="xsd:string"/> 
<xsd:element name="footer" type="xsd:string"/> 
<xsd:element hame="undefined" type="xsd:strlng"/> 
<xsd:element name="copyright" type="xsd:strlngV> 
<xsd:element name="rightHeader" type="xsd:string"/> 
<xsd:element name="rightColumnText" type="xsd:string"/> 
</xsd :complexType> 
</xsd:eiement> 
</xsd: schema > 



The schema includes ''elements" that correspond to the blocks of text within the document 
400. For example, the element '"title" corresponds to the title 402 and the element 
"author" corresponds to the author information 404. 
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[0046] A portion of the model file associated with the schema of Table 2 is 
illustrated In Table 3. In an exemplary embodiment, the model file, which references the 
schema file, i.e., twoColumn.xsd, Is trained from a collection of documents. The model file 
contains the physical characteristics of each element within the schema, their spatial 
relationships, and the relative weight of the characteristics and spatial relationships. 



Table 3 

<?xml version="l.O"?> 

<!-- Created by j zegmdlWr iteXml at 12:22:49 on Friday, 19 April 2002 --> 
<jzegGRAPH class="dociinient" n\iranode="10" th="20" nproba"0"> 

<scheniainf o xsi : schemaLocation»" twoColumn.xsd" 
xmlns : xs i= " ht tp : / /www . w3 . org/ 2 OOl/XMLSchema- instance " / > 

<jzegNODE id:=" 0" pos=" 238 862 1249 1477 743 1169 1012 615" wt=" 

2 13 0 3 15 0" wnull=" 80000" 
label = " abstract /> 

<jzegNODE id=" 1" pos=" 673 409 1920 766 1296 587 1247 357" wt«" 
0 2 0 12 10 1" wnull=" 80000" 

label = " author " / > 

<jzegNODE id=" 2" pos=" 198 2667 1222 2971 709 2819 1024 304" wt=" 

3 1 3 1 3 1 10 14" wnull=" 80000" 
IsQDel- " copyright " / > 

<jzegNODE id=" 3" pos=" 2021 2961 2361 2992 2191 2976 340 31" wt=" 

0 0 0 0 0 0 0 0" wnull=" 588" 
label= "footer" /> 

<jzegNODE id=" 4" pos=" 236 1529 1252 2627 744 2078 1016 1099" wt=" 
3 0 3 13 15 0" wnull=" 80000" 

labels " lef tColumnText " / > 

<jzegNODE id=" 5" pos = " 160 70 1460 127 810 99 1299 57" wt=" 

2 11 3 9 3 10 13 21" wnull=" 80000" 
labels" lef tHeader"/> 

<jzegNODE id=" 6" pos=" 2382 3097 2430 3136 2406 3116 49 39" wt=" 

3 6 3 7 3 7 9 10" wnull=" 80000" 
label = "page "/ > 

<jzegN0DE id=" 7" pos=" 1325 857 2348 2953 1836 1905 1022 2096" wt=" 
2 13 13 14 1" wnull=" 80000" 

label= " r ightColuranText " /> 

<jzegN0DE id=" 8" pos=» 1932 64 2354 122 2143 93 422 58" wt=" 

1 8 2 9 2 9 1 10" wnull=" 80000" 
label = " r ight Header " / > 

<jzegNODE id=" 9" pos=" 509 220 2073 350 1291 285 1564 130" wt=" 
1 8 1 2 3 3 0 2" wnull=" 80000" 

label="title"/> 

<jzegEDGE idl=" 0" id2=" 0" ov="-l" rel=" 22222131 3" wt=" 
100 100 100 100 100 100 100 100 100"/> 

<jzegEDGE idl=" 0" id2=" 1" ov="-l" rel=" 13131133 3" wt=3" 
100 100 100 100 100 100 100 100 100"/> 



[0047] A portion of an XML layout file resulting from the segmentation of the 
document Image 400 and the conversion of text images to digital text Is Included In Table 
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4. In the illustrated embodiment, the results are stored by text lines and segmented Into 
zones. This file contains coordinates of each zone and the coordinates and contents of 
each line within each zone. (Note: in this example, the font size information is disabled.) 



Table 4 

<PAGE xmlns«"http : //www. re search, peuiasonic . com/PINTIj/physical" 
xsi : schemaLocat ion= " twoColumn . xsd" 

xinlns :xsi="httpr//www.w3 .org/ 20 Ol/XMLSchema- instance "> 

<20NE id="0" box="183 74 1473 127" zone -type=" TEXT" f ont-size="0" > 

<LINB id«"0" box="183 74 1473 127" f ont-size="0" >< ! [CDATA [CHI • 95 MOSAIC 
OF CREATIVITY - May 7-11 1995 ]]> 

</LINE> 
</ZONE> 

<ZONE id="l" box="562 225 2041 466" zone- type= "TEXT" f ont-size=" 0"> 

<IjINE id="0" box="610 225 2012 297" f ont-sizeo"0" ><! [CDATA [High- End High 

School Commiinication: ]]> 
</LINE> 

<LINE id="l" box="583 315 2041 389" f ont-size=" 0" ><! [CDATA [Strategies and 
Practices of Students in a ] ] > 
</IiINE> 

<IiINE id="2" box="562 407 1739 466" f ont-size="0" ><! [CDATA [Networked 
Environment ] ] > 

</LINE> 
</ZONE> 

<ZONE id="2" box="1764 62 2368 133" zone - type = "TEXT" f ont-size="0"> 
<LINE id="0" box«"1803 77 2362 116" f ont-size=" 0" ><! [CDATA [Doctoral 

Consort iiam ] ] > 
</LINE> 

</ZONE> 

<ZONE id="3" box="920 488 1690 833" zone -type- "TEXT" f ont-size=" 0"> 

<IiINE id="0" box="1129 499 1482 546" font-size="0" ><! [CDATA [Barry J. 

Fishman ] ] > 
</IjINE> 

<LINB id="2" box="922 559 1691 609" f ont-size*" 0" ><I [CDATA [School of 
Education and Social Policy ] ] > 
</LINE> 

<LINE id="3" box="1058 620 1553 668" font-size="0" ><! [CDATA [Northwestern 

University ] ] > 
</IiINE> 

<IiINE id="4" box="1100 681 1508 723" font-size= "0" ><! [CDATA [Evans ton, IL 
60208 ]]> 
</LINE> 

<LINE id="5» box="1150 740 1460 785" font-size="0" ><![CDATA[( 708 ) 467 - 
2405 ]]> 
</LINE> 

<LINE id="6" box=:"1044 800 1565 833" font-size="0" 
X ! [CDATA [bfishman®covis .nwu.edu ] ] > 

</IiINE> 
</ZONE> 
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[0048] An XML label file resulting from the labeling of the zones is included In Table 
5. The XML label file references the schema and the layout file. The XML file contains the 
logical association between elements in the schema (by element name) and zones within a 
document layout (by zone number, defined in the layout file). 



Table 5 

<?xml version="1.0"?> 

<!-- Created by j zeglogWriteXml at 16:43:10 on Monday, 07 July 2 003 
<document layout="C: \XMIjConverter\newSeg\test\chi95\chi95o001_layout .xml" 
xsi : schemaLocation= " twoColumn . xsd" 

xmlns :xsi="http: //www. w3 .org/2001/XiyiLSchema-instance" > 

<leftHeader idref="0"/> 

<title idref="l"/> 

< right Header idref="2"/> 

<author idref="3"/> 

<abs tract idref="4"/> 

<lef tColumnText idref="5"/> 

<leftColiininText idref«"6"/> 

<lef tColumnText idref ="?"/> 

<copyright idref="8"/> 

<rightColuninText idref="9"/> 

< right ColumnText idref = "10 "/> 

< right ColumnText idref="ll"/> 

< right ColumnText idref="12"/> 

<page idref="13"/> 
</ document > 



[0049] A portion of a structured document XML file Is include in Table 6. The 
structured document XML file contains only document contents separated by each logical 
elements. As can be seen, one logical element (e.g., leftColumnText) in the schema) can 
have multiple instances (zones), identified by IreflD (zone ID). 



Table 6 

<?xml version="l. 0"?> 

<?xml- stylesheet type= " text/xsl " href = " chi95o001 .xsl " ?> 

<documeiit xmlns : xsi= " ht tp : //www . w3 . org/ 1999/XMIjScheina- instance " 

xsi : noNameBpaceSchemaLocat:ion»" twoColumn. xsd "> 

<title idref="l"><! [CDATA [High- End High School Communication: 

Strategies and Practices of Students In a 

Networked Environment] ] ></title> 

<lef tColumnText idref = "5">< ! [CDATA [KEYWORDS: Media Spaces, Education, 

Coramuni c at i on , 

Design] ) ></ leftColumnText > 

<lef tColumnText idref="6"><! [CDATA [INTRODUCTION 

Classroom are li]ce islands, isolated hom each other cuid the 

world beyond their boundaries. Students enter an enclosed 
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Space and for the next forty to ninety minutes, all interaction is 
confined to the individuals contained within the classroom 
walls* More often theui not, the instructions strategies 
enqployed in classrooms also isolate students from one another. 
Communication is comprised of back-cuid-brth exchanges 
between teacher cind student, and only rarely from student to 
Student. This dissertation studies the deployment of highly 
interactive computer-based communication tools designed to 
break the boundaries that exist in classrooms, with the goal of 
elaborating principles for the effective design and 
implementation of these environments in school settings. 
] ] ></lef tColumnText> 

<lef tColumnText idref ="7 ">< ! [CDATA[The high school classrooms involved in 
this study have been 

augmented with a suite of highly interactive communication 
tools, including electronic mail, Usenet newsgroups, 
asynchronous multimedia notebooks, remote screen -sharing, 
and desktop video teleconferencing. In the CHI community, 
this combination of tools has come to be known as a media 
space [3,1] . Media spaces enable individuals or groups to 
] ] ></lef tColumnText > 

<abstract idref="4"><! [CDATA [ABSTRACT 

This paper describes a study of the design of computer-based 
communication and media space environments that support 
highly interactive school-based learning communities. The two 
basic questions posed in this research are: (1) How are media 
space tools used by students in these classrooms, both in terms 
of the structure of communications axctivity and the surrounding 
physical and temporal constrains of the environment?; and (2) 
What are possible explanations for student behaviors and 
attitudes with regard to media space tools? The answers to 
these questions will provide insight for the design of next- 
generation media spaces for educational settings. 
] ] ></abstract> 

<author idref="3"><! [CDATA [Barry J. Fishman 
School of Education and Social Policy 
Northwestern University 
Evanston, XL 60208 
( 708 ) 467 -2405 
bf ishman®covis .nwu.edu] ] > 
</author> 

[0050] A portion of a structured document XSL file is Included In Table 7. The 
Structured document XSL file describes how each zone in the structured document XML file 
should be presented (coordinates, font size, etc.). In an exemplary embodiment, this file 
is automatically generated to reflect the original layout of the document. However, it can 
be modified to adapt to different display devices. For example. In an XML browser on a 
PDA, because of the limited display size, the font may be set to a smaller size and/or only 



the '"abstract" element may be displayed. 
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Table 7 

<?xnil version="1.0" eiicoding="gb2312"?> 

<xsl : stylesheet xmlns :xsls="http: //www, w3 .org/1999/XSL/Transform" 

versions= " 1 . 0 " > 

<xsl : template match = "/"> 

<html > <body > 

<xsl : f or-each select =" /document /title"> 

<div id="layerl" style= "position: absolute; width: 1499px; height : 2 61px; z- 

index:l; left: 552px; top: 215px"> 

<Font style= " font -si ze : 25pt ; color : #000000 " > 

<xsl : value-of select ="text{)"/> 

</Font></div> 

</xsl : f or-each> 

<xsl : f or-each select =" /document /leftColumnText"> 
<xsl:if test="@idref [.='5*] "> 

<div id="layer2" style= "position: absolute; width: 1034px; height : 116px; z- 

index:2; left: 247px; top: 1512px"> 

<Font style="font-size: 25pt;color:#000000"> 

<xsl : value -of select =5"text()"/> 

< / Font > < /di v> 
</xsl : if > 

<xsl:if test="@idref [.= ' 6 » ] "> 

<div id="layer2" style= "position : absolute ; width: 1057px; height : 708px; z- 

index:2; left: 236px; top: 1660px"> 

<Font style= "font-size : 25pt ; color : #000000 " > 

<xsl : value-of select ="text()"/> 

</Font></div> 

</xsl : if > 

<xsl:if test="®idref [,=»7'] "> 

<div id="layer2" style= "position: absolute; width: 1045px; height :364px; z- 

index:2; left: 236px; top: 2404px"> 

<Font styles " font-size : 25pt ; color : #000000 "> 

<xsl : value -of select =s"text()"/> 

< / Font >< / di v> 
</xsl : if > 
</xsl : f or-each> 

<xsl : f or-each select = "/document /abstract "> 

<div id=" layers" style= "position: absolute; width: 1044px; height : 6 llpx; z- 

index:3; left: 244px; top: 871px"> 

<Font style=5"f ont- size: 2 5pt; color :# 000 000" > 

<xsl : value -of select »"text()"/> 

</Font></div> 

</xsl : f or-each> 

<xsl : f or-each select =" /document /author "> 

<div id="layer4" styles "position: absolute; width:790px; height :364px; z- 

index:4; left: 910px; top: 478px"> 

<Font style=" font -size : 25pt; color : #000000 "> 

<xsl rvalue -of select s="text()"/> 

</Font></div> 

</xsl : f or-each> 
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[0051] Although the invention has been described in terms of a document processor 
102, labeler 116, and structured document generator 120, It is contemplated that the 
invention may be implemented in software on a general purpose computer (not shown). 
In this embodiment, one or more of the functions of the various components may be 
implemented in software that controls the general purpose computer. This software may 
be embodied in a computer readable carrier, for example, a magnetic or optical disk, a 
memory-card or an audio frequency, radio-frequency, or optical carrier wave. 

[0052] Although the Invention Is Illustrated and described herein with reference to 
specific embodiments, the Invention is not intended to be limited to the details shown. 
Rather, various modifications may be made in the details within the scope and range of 
equivalents of the claims and without departing from the invention. 
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What is Claimed: 

1 1. A metiiod for generating structured docunnent files from a document 

2 image, the method comprising the steps of: 

3 segmenting the document image into one or more zones, at least one of the 

4 one or more zones containing a respective text Image; 

5 converting the respective text images within the at least one of the one or 

6 more zones to digital text; 

7 automatically identifying layout information for each of the one or more 

8 zones; 

9 labeling each of the one or more zones in accordance with a schema; and 

10 automatically associating mark-up language tags with the labeled zones to 

11 generate the structured document files responsive to the identified layout information and 

12 a model file. 

1 2. The method of claim 1, wherein the model file is associated with the 

2 schema and wherein the labeling step comprises at least the steps of: 

3 automatically labeling each of the one or more zones responsive to the 

4 model file. 

1 3. The method of claim 1, further comprising the steps of: 

2 receiving editing commands corresponding to the one or more zones; and 

3 updating the one or more zones responsive to the editing commands. 

1 4. The method of claim 3, wherein the step of receiving editing 

2 commands includes the step of receiving text editing commands and the step of updating 

3 the one or more zones includes the step of editing the digital text responsive to the text 

4 editing commands. 



1 

2 



5. The method of claim 3, wherein the step of receiving editing 
commands includes the step of receiving segmenting commands and the step of updating 
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the one or more zones Includes the step of updating characteristics of the one or more 
zones responsive to the segmenting commands. 

6. The method of claim 1, further comprising the step of: 
receiving editing commands corresponding to the schema; 
updating the schema responsive to the editing commands. 

7. The method of claim 1, wherein the respective text Images are 
displayed on a graphical user interface (GUI) and wherein the converting step comprises at 
least the step of: 

overiaying the respective text images displayed on the GUI with the at least 
one of the one or more zones with the corresponding digital text. 

8. The method of claim 1, wherein the structured document files Include 
an XML file and an XSL file for each document image and wherein the generating steps 
comprises at least the step of: 

formating the XSL file such that information corresponding to each of the 
labeled zones in the XI^L file is displayed In multiple layers on a web browser. 

9. The method of claim 1, wherein the steps of segmenting, converting, 
labeling, and automatically associating marl<-up language tags are performed sequentially 
responsive to a selection of a workflow icon of a graphical user interface and wherein the 
method further comprises the step of: 

updating the workflow icon to represent a next step of the segmenting, 
converting, labeling, and automatically associating mark-up language tags to be 
performed, wherein the workflow icon presents a unique image corresponding to each 
step. 

10. A system for generating structured document files from a document 
image, the system comprising: 



means for segmenting the document image into one or more zones, at least 
one of the one or more zones containing a respective text image; 
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5 means for converting the respective text images within the at least one of 

6 the one or more zones to digital text; 

7 means for automatically identifying layout information for each of the one or 

8 more zones; 

9 means for labeling each of the one or more zones in accordance with a 

10 schema; and 

11 means for automatically associating mark-up language tags with the labeled 

12 zones to generate the structured document files responsive to the identified layout 

13 information and a model file. 

1 11. The system of claim 10, further comprising : 

2 means for receiving editing commands corresponding to the one or more 

3 zones; and 

4 means for updating the one or more zones responsive to the editing 

5 commands. 

1 12. The system of claim 10, further comprising: 



2 { means for receiving editing commands corresponding to the schema; 

3 means for updating the schema responsive to the editing commands. 

1 13. A structured mark-up language generator for generating structured 

2 document files from a document image, the generator comprising: 

3 a document processor that: a) segments the document image Into one or 

4 more zones, at least one of the one or more zones containing a respective text image, b) 

5 identifies layout information for each of the one or more zones, and c) converts the 

6 respective text images within the at least one of the one or more zones to digital text; 

7 a labeler that labels each of the one or more zones in accordance with a 

8 schema; and 
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9 a structured document generator that generates the structured document 

10 files responsive to the identified layout information and a model file. 

1 14. The generator of claim 13, further comprising: 

2 an editor coupled to the document processor that enables editing of the 

3 digital text and the one or more zones. 

1 15. The generator of claim 13, further comprising: 

2 an editor coupled to the labeler that enables editing of the labels for each of 

3 the one or more zones. 

1 16. A graphical user interface (GUI) for generating structured document 

2 files from a document image, the GUI comprising: 

3 a document panel for displaying a document Image; 

4 a schema panel for displaying a schema corresponding to the document 

5 Image; and 

6 a worl<flow Icon for directing the generation of at least one structured mark- 

7 up language document from the document image, the workflow icon reflecting a next step 

8 in a process to generate the at least one structured mark-up language document. 

1 17. The GUI of claim 16, wherein the process includes sequentially 

2 performing the steps of loading an image, segmenting the image into zones, converting 

3 text within the zones to digital text, labeling the zones, and generating the at least one 

4 structured document and wherein the workflow Icon is updated during the process to 

5 present unique Images corresponding to each step to be performed in the process. 

1 18. A computer readable medium including software that Is configured to 

2 control a general purpose computer to implement a method for generating structured 

3 document files from a document image, the method comprising the steps of: 

4 segmenting the document image into one or more zones, at least one of the 

5 one or more zones containing a respective text Image; 
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converting the respec±ive text Images within the at least one of the one or 
more zones to digital text; 

automatically Identifying layout information for each of the one or more 

zones; 

labeling each of the one or more zones In accordance with a schema; and 

automatically associating mark-up language tags with the labeled zones to 
generate the structured document files responsive to the identified layout information and 
a model file. 

19. The computer readable medium of claim 18, wherein the method 
implemented by the soft:ware configured general purpose computer further comprises: 

updating the one or more zones responsive to editing commands 
corresponding to the one or more zones. 

20. The computer readable medium of claim 18, wherein the method 
implemented by the software configured general purpose computer further comprises:. 

updating the schema responsive to editing commands con-esponding to the 

schema. 
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High-End High School Communication: 
Strategies and Practices of Students in a 
Networked Environment 



School of Education and Social Policy 
Northwestern University 
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ABSTRACT 

This paper describes a study of the design of computer-based 
oommunication and media space environments that suppoit 
highly interactive school-based learning communities. The two 
basic questions posed in this research are: (1) How are media 
space tools used by students in these classrooms, both in terms 
of the stiucture of commanications activity and the sunoumting 
f^ysical and temporal constrmnts of the environment?; and (2) 
What are possible explanations for student behaviors and 
attitudes with regard to media space tools? The answers to 
these questions will provide insight for the design of next- 
generation media spaces for educational settings. 

KEYWORDS: Media Spaces, Education, Communication, 
Design 

INTRODUCTION 

Classrooms are like islaads^ isolated from each other and the 
world beyond their boundaries. Students enter an enclosed 
space and for the next forty to ninety minutes, aU interaction is 
confined to the individuals contained within the classroom 
walls. More often than not, the instructional strategies 
employed in classrooms also isolate students from one another. 
Communication is comprised of back-and-forth exchanges 
between teacher and student, and only rarely from student to 
student. This dissertation studies the deployment of highly- 
interactive computoc^based communication tools designed to 
break the boundaries Uiat exist in classrooms, wiUi the goal of 
elaborating principles for the effective design and 
implementation of these environments in school settings. 

The high school classrooms involved in this study have been 
augmented with a suite of highly interactive communication 
tools, including electronic mail, Usenet newsgroups, 
asynchronous multimedia notebooks, remote screen-sharing, 
and desktop video teleconferencing. In the CHI community, 
this combination of tools has come to be known as a media 
space [3, 1]. Media spaces enable individuals or groups (o 

Pemtisslon to copy without fee all or part of this material Is 
granted, provided that the copies are not made or distributed for 
direct oommerdai advantage, the ACM copyright notice and the 
title of the publication and its date appear, and notice is given 
tbal copying Is by permission oi ACM. To copy otherwise, or to 
repubteh, requUes a fee and/or specitlc permission. . 
CHi' Companion 95. Denver, Colorado, USA 
^ 1995 ACM O-B9791-75S-3/95/0005..43.50 



work together, even when dxey cannot be temporally or spatially 
co-located. This makes it possible for students to collaborate 
t>oth with their distant peers and with experts outside the 
classroom. It is argued that classrooms equipped with media 
space technologies will become the norm over the next decade, 
thus making it important to understand student behaviors and 
attitudes with respect to the use of these communication 
technologies. 

Two basic questions are asked in this reseaicfa: (1) How are 
media space tools used by students in these classrooms, both 
In terms of the structure of communications activity and the 
surrounding physical and temporal constraints of the 
environment; and (2) What are possible explanations for student 
behaviors and attitudes with regard to media s|$ace tools? The 
answers to these questions m intended to provide guidance 
for the design of next-generation implementations of classrooms 
where Ydgjtily interactive conununication tools are used to link 
students as peer collaborators to professional communities, 
teachers, and othor students both witiiln and beyond the walls 
of the school. 

BACKGROUND TO THIS WORK 

AtCHI*94 in Boston, a panel entitled **Media Spaces andTheir 
Application in K- 12 and College Learning Communities" [5] 
presented three different educational projects that employed 
varying configurations of media space technology to enhance 
learning environments. An Important result of tltis panel was 
the recognition that the needs of school environments are 
significantly different than the white collar settings in which 
most media space research is conducted. Each of the tfiree 
projects represented on the panel employed a different solution 
(both technological and pedagogical) to the problem of cross- 
classroom and beyond-school conununication, but there was 
litde empirical evidence to support any of the approaches 
presented. 

In a search for past research to inform the question of media 
space application in classrooms, two distinct, but related, 
literatures were identified. The first, body of research is 
comprised of studies describing the use of various 
communication tools in educational and work settings, 
primarily in terms of message flow over time [61; studies of 
the classroom as a work environment enabled by 
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METHOD, SYSTEM, AND APPARATUS FOR GENERATING STRUCTURED DOCUMENT FILES 

CROSS REFERENCE TO RELATED APPQCATIONS 

[0001] This application Is related to and daims the benefit of U.S. Provisional 
Application No. 60/404,581, filed August 20, 2002, for ''A System for Genei^tlng 
Structured Document." In addition, this application Is related to U.S. Patent Application 
No. 10/293,859 , filed November 13, 2002, for ""Document Classification and Labeling 
Using Layout Graph Matching." 

RELD OF THE INVENTION 

[0002] The present Invention relates to the field of structured languages and, more 
particularly, to the generation of structured language document files from document 
Images. 

BACKGROUND OF THE INVENTION 

[0003] Structured languages such as extensible mark-up language (XML) enable 
the creation of structured document files that are easily searchable and are viewable 
across multiple platforms, e.g., on a desktop computer and on a cellular telephone. For 
example, a structured document file retrieved via a global Information network (e.g., the 
Intemet) can be viewed in full on a desktop computer and can be viewed as text only on a 
cellular telephone. It is often desirable to convert existing hard copy documents or images 
of documents to structured document files to facilitate searching and displaying these 
documents. Accordingly, methods, systems, and apparatus for converting documents to 
structured document files are useful. 

[0004] Existing documents are typically converted to structured document files by 
scanning the documents and automatically converting the text within the scanned 
documents to digital text using optical character recognition (OCR) software. The scanned 
and converted documents are then formatted, either manually or using proprietary data 
structures, to add mark-up language tags. Ofiren, several different software packages are 
employed to perform each of these steps. These methods for generating structured 
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document files tend to be Inflexible, time consuming, and/or difficult to use. In addition, 
the original formatting of the document is often lost, e.g., font sizes, emphasis, etc., 
malcing them more difficult to read when they are displayed. 

[0005] Accordingly, methods, systems, and apparatus for converting existing 
documents to structured document files are needed that are not subject to the above 
limitations. The present invention fulfills this need among others. 

SUI^MARY OF THE INVEIVJTION 

[0006] The present invention is a method, system, and apparatus for generating 
structured document files from document images. Structured document files are 
generated by segmenting the document image into one or more zones containing 
respective text Images, converting the respective text images to digital text, automatically 
identifying layout information for each of the one or more zones, labeling each of the one 
or more zones in accordance with a schema, and automatically associating maric-up 
language tags with the labeled zones to generate the structured document files responsive 
to the identified layout information and a model file. 

BRIEF DESCRIPTION OF THE DRAWINGS 

[0007] Rgure 1 is a block diagram that conceptually represents an exemplary 
system architecture for generating structured document files from document images in 
accordance with the present invention; 

[0008] Rgure 2 is a flow chart of exemplary steps for generating structured 
document files from document images in accordance with the present invention; 

[0009] Figure 3 is an exemplary graphical user interface (GUI) for assisting a user 
in generating structured document files in accordance with the present invention; and 

[0010] Figure 4 is an exemplar/ document from which structured document files are 
generated In accordance with the present invention. 
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DETAILED DESCRIPTION OF THE INVENTION 

[0011] RG. 1 is a conceptual representation of an exemplary system architecture 
100 for generating structured document files from document Images In accordance with 
the present Invention. One or more blocks within the illustrated system architecture 100 
can be performed by the same piece of hardware or module of software. It should be 
understood that embodiments of the present invention may be implemented in hardware, 
software, or a combination thereof. In such embodiments, the various component and 
steps described below would be Implemented in hardware and/or software. 

[0012] In the illustrated system architecture 100, an electronic image of a 
document (the "document image'O is applied to a document processor 102. In certain 
exemplary embodiments, the document image Is generated by scanning a physical 
document using conventional scanning techniques. In certain other exemplary 
embodiment, the document image is supplied In an electronic format such as a Tagged 
Image File Format (tiff) file. Joint Photographic Experts Group (jpeg) file, or other such file. 
In these embodiments, a format converter (not shown) may be used to convert the 
document image Into a format compatible with the present Invention. Suitable document 
Images and format converters for use with the present invention will be readily apparent to 
those of skill In the related arts. 

[0013] The document processor 102 processes the document image In preparation 
for labeling and generating the structured document file(s), which actions are described in 
greater detail below. The illustrated document processor 102 includes a segmenter 104, a 
text converter 106, and a zone and text editor 108. The segmenter 104 segments the 
document image into zones containing text or images. For example, the segmenter 104 
may create a zone containing the title of a document, a zone containing a paragraph 
within the document, and a zone containing a figure. In addition, the segmenter 104 
determines layout information for the zones. For example, the font size and the position of 
the zone on the document. A suitable segmenter for use with the present invention will be 
readily apparent to those of skill in the art of Image processing. Additional Information 
regarding segmenters can be found in commonly assigned U.S. Patent Nos. 5,892,843 and 
6, 327,388 to Zhou et al. entitled "*Title, Caption and Photo Extraction from Scanned 
Document Images" and "^Identification of Logos from Document Images,'' respectively. 
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[0014] In an exemplary embodiment, the segmenter 104 identifies which zones 
contain text images and which zones contain figures. In certain exemplary embodiments, 
each zone is displayed in a color that represents the type of information within that zone. 
For example, text image may be displayed in one color, e.g., red, and non-text images 
such as tables and figures may be displayed in another color, e.g., green. In certain other 
exemplary embodiments, the zones may be distinguished in other ways such as with a 
border having a different color or pattem. 

[0015] The text converter 106 converts the text Images of the zones to digital text, 
i.e., text which Is searchable and editable. For example, the text converter may convert 
the letters with the text Images to their ASCII equivalent. In a exemplary embodiment, 
the text converter is a conventional optical character recognition (OCR) software tool. 
Suitable text converters for use with the present invention will be readily apparent to those 
of skill in the art of image processing. 

[0016] The zone and text editor 108 edits the zones and the digital text. In an 
exemplary embodiment, the zone and text editor 108 may add zones, delete zones, or 
change the size of individual zones responsive to user inputs. For example, a user may 
enlarge a zone containing a portion of a document title to include the entire title. In an 
exemplary embodiment, layout information associated with a zone is updated in 
accordance with the changes to the zones. The zone and text editor 108 also may change 
the digital text responsive to user inputs. For example, misspelled words may be corrected 
by a user. In an exemplary embodiment, the zone and text editor 108 receives user inputs 
via a graphical user interface, which is described in detail /below. Suitable zone and text 
editors for use with the present invention will be readily apparent to those of skill In the art 
of Image processing. 

[0017] The document, as processed by the document processor 102, is applied to a 
model selector 110. The model selector 110 selects a previously developed model file, 
described below, having features that resemble features of the document. In an 
exemplary embodiment, the model selector selects the model file from a plurality of 
previously developed model files. Each of the model files references a schema, which 
describes the structure of a document that contains valid semantics (e.g. title, author, 
abstract etc. for a document such as a technical paper) and includes physical 
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characteristics for the elements of the schema and their spatial relationships relative to 
one another. 

[0018] In an exemplar/ embodiment, the model file is selected by a user, e.g., via 
the graphical user Interface (GUI) described below. In an alternative exemplary 
embodiment, the model selector 110 compares features of the processed document Image 
to stored features of previously developed model flies to automatically select a model file. 
In certain exemplary embodiments, a costing technique is employed with a cost assigned 
to each feature and lower costs representing a higher level of resemblance. In accordance 
with this embodiment, a comparison cost is determined for eadi available model file 
compared to the document image and the model file with the lowest cost Is selected. A 
method for automatically selecting a model file by matching features Is described In 
commonly assigned U.S. Patent Application No. 10/293,859 , filed November 13, 2002, for 
^Document Classification and Labeling Using Layout Graph Matching." having at least one 
common Inventor (referred to herein as the '^Document Classification and Labeling 
Application'^. 

[0019] A schema editor 112 edits the schema. In an exemplary embodiment, the 
schema is retrieved based on a reference to the schema in the model file. In an 
alternative exemplary embodiment, the schema may be referenced by a user, e.g., via the 
GUI described below. The schema editor 112 may be used to add or remove elements 
from the schema responsive to user inputs. In an exemplary embodiment, the^schema 
editor 112 is displayed in a tree-view and the user inputs are received via the GUI 
described below. A suitable schema editor will be readily apparent to those of skill in the 
related arts. 

[0020] A model developer 114 develops the models for use by the model selector 
110. In an exemplary embodiment, the model developer 114 develops the model by 
processing document samples. In certain exemplary embodiment, the model developer 
114 develops the model responsive to user inputs. If the schema is changed by the 
schema editor 112, the model developer 114 needs to develop a new model in accordance 
with the new schema that accommodates the new relations. A suitable model developer 
for use with the present Invention Is described in the Document Classification and Labeling 
Application. 
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[0021] In an exemplary embodiment, models are developed at a system level. 
When developed at the system level, a user's edit and correction activities of logical 
labeling results are monitored. An automatic model learning process updates the 
document model through a feedback loop based on user modified results. In an 
altemative exemplar/ embodiment, models are developed at the user level. When 
developed at the user level, a GUI tool is provided to allow a more knowledgeable user to 
manually create a new model from a set of known samples. 

[0022] The document, as processed by the document processor 102, is also applied 
to a labeler 116. The labeler 116 applies labels to the zones defined by the document 
processor 102 in accordance with the schema. For example, the labeler may label a zone 
containing the title of the document with the element "title." In an exemplary 
embodiment, the labeler applies labels to the zones responsive to a document model 
selected by the model selector 110. 

[0023] In an exemplary embodiment, the labeler 116 automatically labels the zones 
using a layout graph technique. An exemplary layout graph represents each schema 
element associated with a selected model file and Its spatial relationships to one or more of 
the other schema elements and another exemplary layout graph represents each zone In a 
document Image and its spatial relationship to one or more of the other zones. In the 
exemplary embodiment, a document image Is compared to a selected model by the layout 
graphs using a known global scale over total cost matching technique. Because some 
elements In a document may correspond to multiple zones, multiple zones may match the 
same element. A suitable layout graph technique for use with the present Invention, from 
which one skilled In the art can develop a suitable labeler 116, is described in the 
Document Oasslfication and Labeling Application. 

[0024] A label editor 118 enables manual editing of the labeled zones. In an 
exemplary embodiment, the label editor 118 updates the labels on zones applied 
automatically by the labeler 116 responsive to user Inputs. For example. If the labeler 116 
labeled a zone containing the title of the document with the element "author," the label 
editor can be used to change the label of that zone to the correct element. I.e., "^title." In 
an altemative exemplary embodiment, the label editor 118 labels each of the zones 
manually responsive to user Inputs. In an exemplary embodiment, the label editor 118 
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receives user Inputs via the GUI described below. A suitable label editor 118 for use with 
the present invention will be readily apparent to those of skill in the art of image 
processing. 

[0025] A structured document generator 120 generates structured document files 
responsive to layout Information associated with the zones, labeling results, and the 
selected model file. In an exemplary embodiment, the structured document generator 120 
generates an extensible mark-up language (XML) file and a extensible style-sheet 
language (XSL) file for each document image that it processes. The XML file represents 
the document structure and the XSL file represents the document layout. In an exemplary 
embodiment, the XSL file may represent layout information such as font type and size, 
font color, and zone coordinates. 

[0026] To develop the XML file, the exemplary structured document generator 120 
receives layout information from the document processor 102 and labeling results from the 
labeler 116. In an exemplary embodiment, the layout Information contains the number of 
zones within the document, Identification numbers for each zone, and the location of each 
zone. In addition, the structured document generator 120 receives digital text for each 
zone containing a text image from the document processor 102. In certain exemplary 
embodiments, the document processor 102 develops a layout file that includes the layout 
Information and the digital text. In these embodiments, the document processor 102 
passes the layout file to the structured document generator 120 for processing. In certain 
other exemplary embodiments, the digital text is Included within the labeling results. 

[0027] The exemplar/ structured document generator 120 uses the labeling results 
to match each zone to the appropriate schema elements. The structured document 
generator 120 then combines the layout file and the labeling results in a manner that will 
be readily apparent to those skilled In the art of computer programming to generate the 
XML file. A portion of an exemplar/ XML file Is depicted In Table 6 below. 

[0028] In certain exemplar/ embodiments for generating the XML file, the 
structured document generator 120 also receives the model file, which contains the 
schema, from the model selector 110. The document generator 120 may then validate the 
labeling results by comparing the labeling results to the schema to verify that each label of 
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the labeling results corresponds to a schema element. In addition, tiie structured 
document generator 120 may use the model file to incorporate a complete document tree 
structure into the XML file. For example, the element "^name" may contain two sub- 
elements, e.g., first name and last name. In this embodiment, the structure for the sub- 
elements may be Included In the XML file. The incorporation of the document tree 
structure into the XML file will be readily apparent to those of skill in the art of computer 
programming. Also, the structured document generator 120 may use the mode! file to 
match individual elements to corresponding layout information In the layout file, e.g., 
using zone coordinates contained in the layout file and in the model file. 

[0029] To develop the X5L file, the exemplary structured document generator 120 
receives the layout Information from the document processor 102, the labeling results from 
the labeler 116, and the model file from the model selector 110. Pseudo'code to direct 
element processing to generate the XSL file is depicted in Table 1. 



Start root of tree 
Repeat nodes 
If leaf node; no child node 
if this node matches multiple zones 

output xsl template using <xsl:for-each> 
else 

output xsl template using <xsl:template match> 

get next node 
Else; has child node 

get child node 
Endlf 



The pseudo code depicted in Table 1 illustrates the processing of the elements by the 
structured document generator 120. In a tree view representation of the schema, each 
element of the schema Is represented as a node. Each node can have one or more child 
nodes. For example, a logical element '^author^ can have two child nodes, e.g., "^last 
name" and "first name", and it can have multiple Instances to reflect multiple authors. A 
node can also be a leaf node, which indicates there is no branches from this node, such as 
"^rst name" or^last name.'' Processing continues until all elements/nodes are processed. 



TABLE 1. 
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[0030] For each element processed by the structured document generator 120, the 
structured document generator 120 matches the element to corresponding layout 
Information in the layout file, e.g., using zone coordinates contained in the layout file and 
In the model file. The structured document generator 120 then combines the element with 
the corresponding layout information to generate the XSL file in a manner that will readily 
apparent to those of skill in the art of computer programming. 

[0031] In certain exemplary embodiments, a layer concept associated with the 
hyper text mark-up language (HTML) preserves the original layout, e.g., using 
<DIV></DIV> tags In the XSL file. Each layer enclosed within the <DIV></DIV> tags is 
independent of every other layer. Thus, a zone in one layer has no effect on the position 
of a zone In another layer when the zones are displayed on a known web browser (not 
shown). Accordingly, a zone may be assigned coordinates with respect to a common origin 
for display on a web browser without affecting the positioning of any other zone. In 
addition, each zone can have its own style, e.g., font size, type, and color. In an 
exemplary embodiment, each zone Is assigned to a different layer. The original 
coordinates for each zone are then used to develop display coordinates in a known manner 
to display the zone on a web browser. Since the original coordinates for the zones are 
used to position the zones, the zones are referenced to a common origin, and the zones do 
not affect the position of zones in other layers, the position of the zones when displayed on 
a web browser will at least partially match the original layout of the original document 
Image when all layers are displayed. Style information such as font size may also be 
included to increase the resemblance between the displayed document and the original 
document image. A portion of an exemplary XSL file is depicted In Table 7 below. 

[0032] In certain exemplary embodiments, one or more of the zones may contain 
non-text Images (not shown) that are not converted to digital text such as graphs, 
.pictures, etc. In an exemplary embodiment, for each zone containing a non-text image 
the structured document generator 120 generates an image file from the portion of the 
original image within a zone. The structured document generator 120 then inserts a link 
to the image file in the XML file in a manner similar to the Insertion of digital text 
described above to generate the XML file. In addition, the structured document generator 
120 generates the XSL file in a similar manner as described above for text images with the 
exception that style information such as font size Is not included. 
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[0033] RG. 2 depicts a flow chart 200 of exemplary steps for generating structured 

document files in accordance with the present invention. Processing begins at block 202 

with the segmentation of the document image into zones at block 204. At block 206, text 

Images within the zones are converted to digital text. At block 208, the zones and digital 

text are edited. In an exemplary embodiment, the zones are segmented, digital text is 

converted, and zones and digital text are edited as described above with reference to the 

segmenter 104, text converter 106, and editor 108, respectively, of RG. 1. 

« 

[0034] At block 210, layout information for the document image is identified. The 
layout information includes non-content related features that define the look of the 
document. These features may Include, by way of non-limiting example, font size, 
emphasis formatting, positional Information; etc. In an exemplary embodiment the layout 
information is used in the generation of the structured document flies such that a 
displayed Image of the structured document files retains at least a portion of the original 
layout information associated with the document image. Because the original layout 
information Is maintained, the displayed images reflect the formatting of the original 
documents, thus making them more easy to read. In an exemplary embodiment, the 
layout information is identified by the above-described segmenter 104 (RG. 1). 

[0035] At block 212, the zones are labeled in accordance with a schema and, at 
block 212, mark-up language tags are associated with to the labeled zones to create the 
structured document files. In an exemplary embodiment, the zones are labeled and the 
tags are associated as described above with reference to the labeler 116 and the 
structured document generator 120, respectively, of RG. 1. 

[0036] RG. 3 depicts an exemplary graphical user interface (GUI) 300 for use In the 
present invention. The illustrated GUI 300 includes a tool bar 302, a schema panel 304, 
and a viewing panel 306. The GUI 300 provides an easy to user interface that allows a 
user to generate structured document files from document Images. 

[0037] In an exemplary embodiment, a user accesses a workflow menu (not shown) 
by selecting a ''workflow" indicator 308 from the tool bar 302. In certain exemplary 
embodiments, the workflow menu guides the user sequentially through the structured 
document file generation process described above, e.g., segmenting the document image 
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into zones, converting text to digital text, labeling the zones, and generating the 
structured document'^files. In certain other exemplary embodiments, the user is guided 
through the workflow process by a "^"workflow" icon 310, which is described in detail below. 
In certain exemplary embodiments, arrow Indicators 311 are available to move back and 
forth sequentially through the workflow process. Altemativeiy, the entire workflow process 
of generating a structured document from a document Image Is performed automatically 
by selecting an ""auto execute" Icon 312 in the toolbar 302. 

[0038] The ^workflow" icon 310 displays unique images that correspond to different 
steps of the workflow process. In an exemplary embodiment, the "^workflow" icon 310 
reflects a next step In the workflow process to guide a user sequentially through the 
process of generating structured document files from document images. For example, 
prior to loading a document Image, the "^workflow" icon 310 may display the text *Load 
Image," and after the document image is loaded, but before the document image Is 
segmented, the "^workflow" icon 310 may display the text ""Segment." Selecting the 
workflow icon 310 when the text ""Load Image" Is displayed results in the loading of an 
image and selecting the ''workflow" icon 310 when the text ^*Seg men t" Is displayed results 
in the segmentation of the document image. 

[0039] In an exemplary embodiment, selecting an ""images" icon 314 on the toolbar 
302 Initiates a document image source selection, e.g., via a conventional file open window 
(not shown). A selected document image is then displayed In the viewing panel 306. 

[0040] In an exemplary embodiment, the selection of a document image Initiates a 
model file matching routine that Identifies a model file for the document Image. From the 
identified model file, a schema is Identified for display In the schema panel 304, e.g.. In a 
tree view. Alternatively, a user selects the schema manually by selecting a **schema" icon 
316 on the toolbar 302. In certain exemplary embodiments, the user changes the 
automatically or manually selected schema by selecting the ""schema" icon 316. In certain 
exemplary embodiment, the schema may be updated, e.g., elements may be added or 
removed from the schema, or a new schema may be created using conventional editing 
techniques. Once editing Is complete, the user saves the newly edited (or created) schema 
file. The model matching process, described above, is performed after a new schema is 
saved to select a model con^esponding to the new schema. 
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[0041] 



Document segmentation, text conversion, and labeling are performed in the 



viewing panel 306. In an exerhplary embodiment, the document is segmented and text Is 
converted responsive to the loading of a document image. In alternative exemplar/ 
embodiments, the document is segmented and the text Is converted by selecting the 
"^vorkflow" icon 310 on the toolbar 302 twice (once to Initiate segmentation and once to 
initiate text conversion) or through the workflow menu (not shown) that appears when the 
worlcflow indicator 308 Is selected. In accordance with these embodiments, the document 
Is segmented into '^meaningful" zones according to physical attributes such as font size, 
spacing, etc. In the Illustrated embodiment, segmented zones are displayed with 
bounding boxes overlaid on the original image, which can be corrected by the user using 
conventional techniques. There are several features that are available within the viewing 
panel 306. These features include zoom In/out, zone selection/editing, and zone change 
features. In the illustrated embodiment, text conversion results for identified text regions 
are also overiaid directly in each zone for easy review and editing using conventional 
techniques. It will be readily apparent to those of skill In the art that segmentation and 
text conversion may be performed concurrently or in two distinct steps. 

[0042] After segmentation and text conversion, labels are added to the segments. 
In an exemplary embodiment, labeling is initiated through its selection from the worldlow 
menu or by selecting the "^workflow" Icon 310. In the Illustrated embodiment, the labeling 
results in the display of logical labels on the top left comer of each zone as shown In RG. 
3. In an exemplar/ embodiment, the logical labels can be edited In a conventional 
manner, e.g., by ""right-clicking" to display a pull-down menu (not shown) to link and 
unlink the zone to a schema element or by dragging the schema elements from the 
scheme tree to a zone. In certain exemplary embodiments, the labels associated with the 
zones may be saved by selecting a "^SaveUnk" Icon 318 on the tool bar 302. 

[0043] After labeling, the structured documents are generated by selecting a 
structured document generation indicator in the workflow menu, selecting the "^workflow" 
Icon 310, or selecting a ""Save XML'' Icon 320 on the toolbar 302. In an exemplar/ 
embodiment, this prompts the creation of two structured document files: an XML file and a 
corresponding XSL file. 
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[0044] 



The GUI 300 additionally provides an easy to use interface that allows a user 



to train model files. In an exemplary embodiment, a training mode is entered by selecting 
this mode from the '"woricfiow" menu or by selecting a "LeamModel" icon 322 on the 
toolbar 302. In the training mode, a user edits one or more similar sample documents. 
During editing, the user's edits are monitored and analyzed to develop a model file from 
the sample documents. The new model file can then be used to segment and label 
subsequent documents. 

[0045] RG. 4 depicts a document image 400 to be processed in accordance with the 
present invention. Initially, the document image 400 is scanned using conventional 
scanning software. The Illustrated document Image 400 includes several blocks of text 
including a title 402 and author information 404, e.g., name, telephone number, etc. A 
schema for a two-column text document similar in style to the document image 400 is 
included in Table 2. 



<xsd:schema xmlns:xsd="http://www.w3.org/1999/XMLSchema"> 

<xsd : element name="document'*> 

<xsd:complexType> 

<xsd:element name-"tit!e" type="xsd:string7> 
<xsd:element name="leftColumnText" type="xsd: string "/> 
<xsd:element name="abstract" type="xsd:stringV> 
<xsd:element name="author" type="xsd:string"/> 
<xsd:element name="leftHeader" type=">c5d:stringV>^" 
<xsd:element name="page" type="xsd:stringV> 
<xsd:element name="footer" type="xsd:stringV> 
<xsd:element name=:"undefined" type="xsd:stringV> 
<xsd:element name="copyright" type="xsd:string7> 
<xsd:element name="rightHeader" type="xsd:string"/> 
<xsd:element name="rlghtColumnText" type=''xsd:string"/> 
</xsd:complexlVpe> 
</xsd:eiement> 
</xsd:schema> 



The schema includes ""elements" that correspond to the blocks of text within the document 
400. For example, the element "title" corresponds to the title 402 and the element 
^author^ corresponds to the author information 404. 



Table 2 
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[0046] A portion of the model file associated with the schema of Table 2 is 
illustrated In Table 3. In an exemplary embodiment, the model fije, which references the 
schema file, i.e., twoColumn.xsd, Is trained from a collection of documents. The model file 
contains the physical characteristics of each element within the schema, their spatial 
relationships, and the relative weight of the characteristics and spatial relationships. 



Table 3 

<?xml version="1.0"?> 

<!-- Created by jzegmdlWriteXml at 12:22:49 on Friday, 19 April 2002 --> 
<jzegGRAPH class= "docximent " nuTnnode="10" th="20" nprob«"0"> 

ochemainf o xsi : schemaLocation="twoColvunn.xsd'' 
xmlns : xs i= " ht tp : / /www . w3 . org/ 2001 /XMLSchema- ins t ance " / > 

<jzegNODE id=" O" pos=" 238 862 1249 1477 743 1169 1012 615" wt=" 

2 1 3 0 31 5 0" wnull»» 80000" 

1 absX— " abf?t"i T'ac t " /> 

<jzegN0DE id=" 1" pos=" 673 409 1920 766 1296 587 1247 357" wt=" 
0201210 1" wnull«" 80000" 

label = " author " / > 

<jzegNODE id«« 2" pos«" 198 2667 1222 2971 709 2819 1024 304" wt=" 

3 1 3 1 3 1 10 14" wnull*" 80000" 
1 abel = " copyright " / > 

<jzegNODE id==" 3" pos=" 2021 2961 2361 2992 2191 2976 340 31" wt=" 

0000000 0" Wixull«" 588" 
label =s"footer"/> 

<jzegNODE id=" 4" pos=" 236 1529 1252 2627 744 2078 1016 1099" wt«" 
30313 15 0" wnull«" 80000" 

label = " 1 ef t ColumnText " / > 

<jzegNODE id«" 5" pos=" 160 70 1460 127 810 99 1299 57" wt»" 

2 11 3 9 3 10 13 21" wnull«" 80000" 
labels " 1 eft Header " / > 

<jzegNODE id»" 6" pos=" 2382 3097 2430 3136 *2406 3116 49 39" wt»" ^ 

3 6 3 7 3 7 9 10" wnull«" 80000" 
labels "page "/> 

<j2egNODE id«" 7" pos«" 1325 857 2348 2953 1836 1905 1022 2096" wt-" 
21313 14 1" wnull=" 80000" 

labels" right ColumnText " / > 

<jzegNODE idis" 8" pos=." 1932 64 2354 122 2143 93 422 58" wt«" 

1 8 2 9 2 9 1 10" vniull«" 80000" 
label « " rightHeader " / > 

<jzegNODE id=" 9" pos=" 509 220 2073 350 1291 285 1564 130" wt-" 
18 12 3 3 0 2" wnullo" 80000" 

label="title"/> 

<jzegEDGB idl»" 0" id2=" 0" ov=«"-l" rel«" 22222131 3" wt=" 
100 100 100 100 100 100 100 100 100"/> 

<jzegEDGB idl»" 0" id2«=" 1" ov«"-l" rel=" 13131133 3" wt«" 
100 100 100 100 100 100 100 100 100"/> 



[0047] A portion of an XML layout file resulting from the segmentation of the 
document image 400 and th@ conversion of text Images to digital text Is Included In Table 
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4» In the illustrated embodiment, the results are stored by text lines and segmented Into 
zones. This file contains coordinates of each zone and the coordinates and contents of * 
each line within each zone. (Note: in this example, the font size information is disabled.) 



Table 4 

<PAGE xmlns="http : //www. research. panasonic .com/PINnj/physical" 
xsi : s chemaljoca t ion= " twoColumn . xsd" 

xmlns :xsi«»littp://www.w3 .org/200l/XMliSchema-instance»> 

<ZONE id«"0" box="183 74 1473 127" zone- type« "TEXT" font-si2e«"0"> 

<LINE id«"0" box»"183 74 1473 127" f ont-si2e="0" >< t [CDATA (CHI * 95 MOSAIC 
OP CREATIVITY - May 7-11 1995 ]]> 

</IiINE> 
</ZONE> 

<ZONE id="l" box=.''562 225 2041 466" zone - type = "TEXT" f ont-size="0"> 

<IjINE id="0" box="610 225 2012 297" f ont-size«"0" >< I ECDATA [High-End High 

School Communication: ]]> 
</I*INE> 

<LINE id=="l" box="583 315 2041 389" f ont-size="0" ><! [CDATA (Strategies and 
Practices of Students in a ) ] > 
</IiINE> 

<LINE id«"2" box="562 407 1739 466" font-size="0" ><1 [CDATA [Networked 
Environment ] ) > 

</I.INE> 
</ZONE> 

<ZONE id="2" box«"1764 62 2368 133" zone -type=" TEXT" font-size="0"> 
<LINE id="0" boxa"1803 77 2362 116" f ont-Bize="0" ><1 (CDATA (Doctoral 

Consort iiim ] 3 > 
</liINE> 

</ZONE> 

<ZONE id="3" box«"920 488 1690 833" zone -type- "TEXT" f ont-sizea"0"> 
<LINE id«"0» box«"1129 499 1482 546" f ont-size«"0" >< I [CDATA (Barry J . 

Fishman ] ] > 
</LINE> 

<LINE id«"2" box="922 559 1691 609" font-size=" 0" ><! (CDATA [School of 
Education and Social Policy 3 ] > 

</IiINE> 

<LINE id="3" box="1058 620 1553 668" font-size=»"0" >< I (CDATA [Northwestern 
University ] ] > 
</LINE> 

<IiINE id="4" box«"1100 681 1508 723" font-size="0" >< I [CDATA (Evans ton, IL 
60208 ]]> 
</LINB> 

<LINE id«"5" box="1150 740 1460 785" font-size«"0" ><!(CDATA(( 708 ) 467 - 
2405 ]]> 
</LINE> 

<IjINE id="6" box="1044 800 1565 833" font-size»"0" 
>< ! [CDATA (bf ishmanocovis . nwu . edu ] ] > 

</IiINE> 
</ZONE> 
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[0048] An XML label file resulting from the labeling of the zones is included in Table 
5. The XML label file references the schema and the layout file. The XML file contains the 
logical association between elements in the schema (by element name) and zones within a 
document layout (by zone number, defined In the layout file). 



Table 5 

<?xml versions " 1. 0"?> 

<! — Created by jzeglogWriteXml at 16:43:10 on Monday, 07 July 2003 --> 
<docuinent layout* "C : \XMIjConverter\newSeg\test\chi95\chi95o001_layout .xml" 
xsi : schemaljocat ion=: " twoColumn .xsd" 

xmlns :xsi="http://www.w3,org/2001/XMLScheina-instance" > 

<leftHeader idref="0"/> 

<title idref="l"/> 

<rlghtHeader idref="2"/> 

<author idref="3"/> 

<abs tract idref="4"/> 

<leftColuinnText idref«"5"/> 

<leftColumnText idref="6"/> 

<leftCol\iinnText idref="7"/> 

< copyright idreft3"8"/> 

<right ColumnText idref = 9 " / > 

< right ColuitmText idref =» " 1 0 " / > 

<rightColuinnText idref «"ll"/> 

<rightColuimiText idref = " 12 " / > 

<page idref =** 13" /> 
</ document > 



[0049] A portion of a structured document XML file is include in Table 6. The 
structured document XML file contains only document contents separated by each logical 
elements. As can be seen, one logical element (e.g., leftColumnText) In the schema) can 
have multiple Instances (zones), identified by ireflD (zone ID). 



Table 6 

<?xml version- "1. 0"?> 

<?xna- stylesheet type«"text/xsl" href =»chi95o001.xsl"?> 

<document xmlns :xsi«"http: //www. w3 .org/19 9 9/XMLSchetna- instance" 

xsi : noNamespaceSchemaliOcat ion= " twoColumn . xsd" > 

<title idref«"l"><l [CDATA[High-Bnd High School Coirmunicationi 

Strategies and Practices of Students In a 

Networked Environment] ] ></title> 

< leftColumnText idref ="5" ><! [CDATA [KEYWORDS: Media Spaces, Education, 

Communi c a t i on , 

Design] ] ></ leftColumnText > 

< 1 ef t ColumnTex t i dre f = " 6 " >< ! [ CDATA [ INTRODUCT ION 

Classroom are like islands, isolated horn each other and the 

world beyond their Jaoundaries. Students enter an enclosed 
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Space and for the next forty to ninety minutes, all interaction is 
conf ined to the individuals contained within the classroom 
walls. More often than not, the instructions strategies 
employed in classrooms also isolate students from one another. 

Communication is comprised of back-and-brth exchanges 
between teacher and student, and only rarely from student to 
Student. This dissertation studies the deployment of highly 
interactive computer-based communication tools designed to 
break the boundaries that exist in classrooms, with the goal of 
elaborating principles for the effective design and 
implementation of these environments in school settings. 
] ] ></lef tColumnText> 

<lef tColumnText idref-"7"><! CCDATA[The high school classrooms involved in 
this study have been 

augmented with a suite of highly interactive communication 
tools, including electronic mail, Usenet newsgroups, 
asynchronous multimedia notebooks, remote screen- sharing, 
and desktop video teleconferencing. In the CHI community, 
this combination of tools has come to be Imown as a media 
space [3,1] . Media spaces enable individuals or groups to 
] ] ></lef tColumnText > 

<abstract idref="4"><! [CDATA [ABSTRACT 

This paper describes a study of the design of computer-based 
communication and media space environments that support 
highly interactive school -based learning communities. The two 
basic questions posed in this research are: (1) How are media 
space tools used by students in these classrooms, both in terms 
of the structure of communications axctivity cind the surrounding 
physical and temporal constrains of the environment?; and (2) 
What are possible explainations for student behaviors and 
attitudes with regard to media space tools? The answers to 
these questions will provide insight for the design of next- 
generation media spaces for educational settings. 
] ] ></abstract> 

<author ldref»"3"><l [CDATA [Barry J. Fishman 
School of Education and Social Policy 
Northwestern University . - - 
Evanston, IL 60208 
( 708 ) 467 -2405 
bfishman®covis .nwu.edu] ] > 
< /author > 



Structured document XSL file describes how each zone in the structured document XML file 
should be presented (coordinates, font size, etc.). In an exemplary embodiment, this file 
is automatically generated to reflect the originat layout of the document. However, it can 
be modified to adapt to different display devices. For example. In an XML browser on a 
PDA, because of the limited display size, the font may be set to a smaller size and/or only 
the "^abstract" element may be displayed. 



[0050] 



A portion of a structured document XSL file Is included In Table 7. The 
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Table 7 



<?xinl version="l . 0" encoding="gb2312"?> 

<xsl : stylesheet xmlns :xsl«"http: //www. w3 .org/ 1999 /XSL/Trans form" 

versions " 1 . 0 " > 

<xsl : template match = "/"> 

<html><body> 

<xsl : f or-each select /document /t it le"> 

<div id=:»layerl" style= "position: absolute; width: 1499px; height : 2 61px; z- 

index:!/ left: 552px; top: 215px"> 

<Font style="f ont-size : 25pt /color :#000000"> 

<xsl: value-of select «"text()"/> 

</Font></div> 

</xsl : f or-each> 

<xsl : f or-each select « " /document /left ColumnText " > 
<xsl:if test="®idref [.= '5'] "> 

<div id="layer2" style» "positions absolute; width: 1034px; height :116px; z- 

index:2; left: 247px; top: 1512px»»> 

<Font style="f ont-size: 25pt;color:#000000"> 

<xsl:value-of select a"text()"/> 

</Font></div> 

</xBl:if> 

<xsl:if test="®idref [.=»6'] "> 

<div id="layer2" style= "positions absolute; width: 1057px; height ; 708px; z- 

index:2; left: 236px; top: 1660px"> 

<Font style- "f ont-size:25pt; color; #000000 "> 

<xsl:value-of select «"text()"/> 

</Font></div> 

</xsliif> 

<xsl:if tests'-Oidref [. = »7"3 "> 

<div ids"layer2" style» "position: absolute; width: 1045px; height :364px; z- 

index:2; lefts 236px; top: 2404px''> 

<Font style- " font -size s 25pt ; color : #000000 " > 

<xsl:value-of select a"text{)"/> 

</Font></div> . . y - 

</xsl :if > 

</xsl : f or-each> 

<xsl: f or-each select =" /document /abs tract "> 

<div id=" layers " styles "position: absolute; width: 1044px; height :611px; z- 
index:3; left: 244px; top: 871px"> 

<Pont style="f ont-size: 25pt ;colors#000000»»> ; 
<xsl:value-of select ="text()"/> 
</Pont></div> 
</xsl : f or-each> 

<xsl : f or-each select s** /document/ author ■*> 

<div id«"layer4» styles "i>osition: absolute; width«790px; height :364px; z- 

index:4; left: 910pX; top: 478pX"> 

<Font styleanfont-size:25pt /color :#000000"> 

<xsl:value-of select ="text()"/> 

</Font></div> 

</xsl : f or-each> 
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[0051] Although the invention has been described in terms of a docximent processor 
102, labeler 116, and structured document generator 120, it is contemplated that the 
invention may be implemented in software on a general purpose computer (not shown). 
In this embodiment, one or more of the functions of the various components may be 
implemented In software that controls the genera! purpose computer. This software may 
be embodied In a computer readable carrier, for example, a magnetic or optical disk, a 
memory-card or an audio frequency, radio-frequency, or optical carrier wave. 

[0052] Although the invention Is Illustrated and described herein with reference to 
specific embodiments, the invention is not Intended to be limited to the details shown. 
Rather, various modifications may be made in the details within the scope and range of 
equivalents of the dalms and without departing from the invention. 



wo 2004/019230 



PCT/US2003/026025 



-20- 

What is Claimed: 

1. A method for generating structured document files from a document 
Image, the method comprising the steps of: 

segmenting the document Image into one or more zones, at least one of the 
one or more zones containing a respective text Image; 

converting the respective text images within the at least one of the one or 
more zones to digital text; 

automatically identifying layout information for each of the one or more 

zones; 

labeling each of the one or more zones In accordance with a schema; and 

automatically associating mark-up language tags with the labeled zones to 
generate the structured document files responsive to the identified layout information and 
a model file. 

2. The method of claim 1, wherein the model file is associated with the 
schema and wherein the labeling step compnses at least the steps of: 

automatlcally labeling each of the one or more zones responsive to the 

model file, 

3. The method of dalm 1, further comprising the steps of: 
receiving editing commands corresponding to the one or more zones; and 
updating the one or more zones responsive to the editing commands. 

4. The method of claim 3, wherein the step of receiving editing 
commands includes the step of receiving text editing commands and the step of updating 
the one or more zones Includes the step of editing the digital text responsive to the text 
editing commands. 

5. The method of daim 3, wherein the step of receiving editing 
commands includes the step of receiving segmenting commands and the step of updating 



wo 2004/019230 



PCT/US2003/026025 



-21 - 

the one or more zones Includes the step of updating characteristics of the one or more 
zones responsive to the segmenting commands. 

6. The method of claim 1, further comprising the step of: 
receiving editing commands corresponding to the schema; 
updating the schema responsive to the editing commands. 

7. The method of claim 1, wherein the respective text images are 
displayed on a graphical user Interfece (GUI) and wherein the converting step comprises at 
least the step of: 

overlaying the respective text images displayed on the GUI with the at least 
one of the one or more zones with the corresponding digital text. 

8. The method of claim 1, wherein the structured document files Include 
an XML file and an XSL file for each document Image and wherein the generating steps 
comprises at least the step of: 

formating the XSL file such that information corresponding to each of the 
labeled zones in the XML file fs displayed in multiple layers on a web browser. 

9. The method of claim 1, wherein the steps of segmenting, converting, 
labeling, and automatically associating mark-up language tags are performed sequentially 
responsive to a selection of a workflow Icon of a graphical user Interface and wherein the 
method furt:her comprises the step of: 

updating the workflow Icon to represent a next step of the segmenting, 
convertring, labeling, and automatically associating mark-up language tags to be 
performed, wherein the workflow Icon presents a unique image corresponding to each 
step. 

10. A system for generating structured document files from a document 
Image, the system comprising: 

means for segmenting the document Image into one or more zones, at least 
one of the one or more zones containing a respective text Image; 
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5 means for converting the respective text Images within the at least one of 

6 the one or more zones to digital text; 

7 means for automatically identifying layout Information for each of the one or 

8 more zones; 

9 means for labeling each of the one or more zones in accordance with a 

10 schema; and 

H means for automatically associating mark-up language tags with the labeled 

12 zones to generate the structured document files responsive to the identified layout 

13 Information and a model file. 

1 11. The system of claim 10, further comprising: 

2 means for receiving editing commands corresponding to the one or more 

3 zones; and 

4 means for updating the one or more zones responsive to the editing 

5 commands. 

1 12. The system of dalm 10, further comprising: 

2 i means for receiving editing commands corresponding to the schema; 

3 means for updating the schema responsive to the editing commands. 

1 13. A structured mark-up language generator for generating structured 

2 document files from a document image, the generator comprising: 

3 a document processor that: a) segments the document image Into one or 

4 more zones, at least one of the one or more zones containing a respective text Image, b) 

5 identifies layout information for each of the one or more zones, and c) converts the 

6 respective text images within the at least one of the one or more zones to digital text; 

7 a labeler that labels each of the one or more zones In accordance with a 

8 schema; and 
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9 a structured document generator that generates the structured document 

10 files responsive to the identified layout information and a model file. 

1 14. The generator of claim 13, further comprising : 

2 an editor coupled to the document processor that enables editing of the 

3 digital text and the one or more zones. 

1 15. The generator of claim 13, further comprising : 

2 an editor coupled to the labeler that enables editing of the labels for each of 

3 the one or more zones. 

1 16. A graphical user interface (GUI) for generating structured document 

2 files from a document image, the GUI comprising: 

3 a document panel for displaying a document image; 

4 a schema panel for displaying a schema corresponding to the document 

5 image; and 

6 a workflow icon for directing the generation of at least one structured marl<- 

7 up language document from the document image, the worI<flow icon reflecting a next step 
8... In a process to generate the at least one structured marlc-up language document. 

1 17. The GUI of claim 16, wherein the process Includes sequentially 

2 performing the steps of loading an Image, segmenting the image into zones, converting 

3 text within the zones to digital text, labeling the zones, and generating the at least one 

4 structured document and wherein the worl<flow Icon is updated during the process to 

5 present unique images corresponding to each step to be performed in the process. 

1 18. A computer readable medium including software that is configured to 

2 control a general purpose computer to implement a method for generating structured 

3 document files from a document image, the method comprising the steps of: 



4 

5 



segmenting the document image into one or more zones, at least one of the 
one or more zones containing a respective text image; 
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convertlng the respecUve text Images within the at least one of the one or 
more zones to digital text; 

automatically Identifying layout infonrnation for each of the one or more 

zones; 

labeling each of the one or more zones In accordance with a schema; and 

automatically associating mark-up language tags with the labeled zones to 
generate the structured document files responsive to the identified layout Information and 
a model file. 

19. The computer readable medium of claim 18, wherein the method 
Implemented by the software configured general purpose computer further comprises: 

updating the one or more zones responsive to editing commands 
corresponding to the one or more zones. 

20. The computer readable medium of claim 18, wherein the method 
implemented by the software configured general purpose computer further comprises: 

updating the schema responsive to editing commands corresponding to the 

schema. 
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ABSTRACT 

Tbb papar describes a stndy of ibe de^gn of oon^otBr^based 
communication and media apace cn vir o nm eots timt support 
highly imeiiictlvBschod*l»3ed teaming comiDuiiitics. Thotwo 
baste qaestkw posed hi thb fcseaidi are: (1) How are inedte 
apace toob wed by students in these dassnxnns, both Ja tcnns 
of the soncQne of ooinxinmications scfi viiy and ihe sunouniSog 
jAysicd aiui temporal constraanb of the environment?; and (29 
What are possible explanations for soideat bebavion and 
Qttitiides with regani to media space.tools? The aaswoa to 
these questions will provide In^ght for the design of nc^o- 

KEVWORDS: Media 



woiktogeihei; even when thqr cannot bo tempoiaQy wspatiaDy 
CD4ocated. TOsmaJces it possible ferstodcnls to coDabotwo 
bodi with didr distant peers and with experts outside the 
classiooiiL Ik is aigned that daasiooms equipped with media 
spa ce iechiiologicswin become fteponnoverthe next decade, 
t hns nw ldng it Intportaal to i mc tmm n d stu den t behaviors and 
attStttdes with lespecl to the nse of these communicatioa 



Dm^ooucnoN 

dassiu m i ia are like islands, isolated from each other and the 
woiU beyond their boundaries. Students enter an enclosed 
qiaos and fior the next forty to i&ety minutes, an Intetaction b 
coofined to the individuals contained within the classroom 
walls. More often than not. the inslraetlonal strategies 
employed in clasnnonM alfii>isei1ftt<» <ind<»nt< fipni nTmnffft^hfT. 
Communication Is cum p i isc d of b8ck>and-fortb exchanges 
between teacher and sCDdeaS» and only laidy from student to 
student Tlua dissertation studies the ia^iasmsBl of highly, 
interactive c ompu ter-based f ^ff rmffiw yf*^i*n tools <fe»i g n^ ^ 
break the boundaries that esiist in dassrooms, with the goal of 
elaborating principles for the effective design and 
isqilemeatBtLon of these covinHuaents in sdwol settings. 

The high school classrooms Involved in dds study have been 
angmenied with a suite of higjify Interactive coammnlcadon 
tools, including electronic mail. Usenet newsgroups, 
asjmchronotts multimedia notebooks, zcmote screen-Gfaadnfr 
and desktop video teleconferencing, the CHI codanmtf|y, 
this combination of tools has come to be known as a aerfni 
*poce p, ]]. Media spaces enable faidividinis or ^oqis to 

Pom^laslon to copy wCihout fee aa or part of tMs mataital b 
pantod pnvtdad that tho copies are nsl made or distilbuted for 
woct cammerdal advantaae. the ACM capyiioht nollca and tho 
«wc3lihftpub6caQon and Us data appear, and noQcebdven 
^ ^ctpytng Is by pmmissbn of ACM Tocopy otbenvlso. or to 
recmblsn. requtros a lea andtor spodne pstrmsdon. 
Wr Cpmpanton 95, Denver. Colorado. USA 
A 199SACM O«9791-755-3/95«C0Su43L50 



IXvo basic questiatts are asked iii Ibis icseaidi: (l)Howaie 
media space tools used by sautents in these dassroonis, both 
in terms of the structure of oomnuudeations activity and the 
surrounding physical and temporal constraints of the 
environment; end (2) What are possOde explanations for shident 
behaviors and attitodes with regard to msdiasgiace tools? The 
answers u> these questions are inrrtiidrd to provide pnidawCT 
ffB- the <kdgn of next*geoeialion implementations of classrooms 
where fa^gUy interactive conunumcation toob are used to link 
students as peer collaborators to professional communities, 

teacfaen, and other students both within and beyoad the WBib 
ofifaesdiodL 

BACKGROUND TOTHtS WORK 

At cm '94mBosan.8paad entitled *^(edia Spaces andTheir 
Application hi K-12 and CbDege Leandng Ommmidties*' tS] 
presented three diSerent educational projects that employed 
varying oonfignrafions of media space technology to enhance 
learning environments. Animpoitantfesallof Hdspaiielwas 
the recognition that the needs of school environments are 
sigmficantly different thmi the white collar sett2n&> in which 
most media space research is conducted. Eacb of the three 
projects represented on the panel enjoyed adiffBRot fiolntioa 
(both techaolo^cal and pedagogical) to the problem of cioss- 
dassroom and beyond-school conummlcatiao. bat dxeie was 
Uttle anpirical evidence to sapport aqy ^ Ibe approaches 



In a seardi for past research to inform the questim of media 
space application In classrooms, two dbtioct, but rdated. 
Utecatnres were idenUfied. The first body of research Is 
comprised of studies describiag the use of various 
wnam naication tools In edocational and work settings, 
priaaiily hi terms of message tkm over lime [6]; studies of 
the classroom as a work cBvlronmeac enabted by 
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